Adaptive initial step size selection for simultaneous perturbation stochastic approximation

A difficulty in using Simultaneous Perturbation Stochastics Approximation (SPSA) is its performance sensitivity to the step sizes chosen at the initial stage of the iteration. If the step size is too large, the solution estimate may fail to converge. The proposed adaptive stepping method automatically reduces the initial step size of the SPSA so that reduction of the objective function value occurs more reliably. Ten mathematical functions each with three different noise levels were used to empirically show the effectiveness of the proposed idea. A parameter estimation example of a nonlinear dynamical system is also included

Dimensionality reduction of optimization problems using variance based sensitivity analysis

We propose a new interaction index derived from the computation of Sobol indices. In optimization, interaction index can be used to detect lack of interaction among input parameters. First order interaction indices if they return zero, means that those parameters can be optimized independently holding other parameters constant. Likewise, second order interaction indices can tell if a combination of two parameter can be optimized independently of other parameters. In this way, the original optimization problem may be decomposed into a set of lower dimensional problems which may then be solved independently and in parallel. The interaction indices can potentially be useful in robust optimization as well, since it provides importance measure in minimizing output variances